from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(against_lib="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	1.955136	0.108803	NaN	...	brute	-1	1	0.663	0.343924	0.006344	1.000	5.684795	5.685762	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	0.022132	0.002733	NaN	...	brute	-1	1	1.000	19.093011	0.076671	0.757	0.001159	0.001159	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	22d5eac4a4146149b35eae6914921514	2cf49829391aa7891e877f2ed070adf0	2.860174	0.070873	NaN	...	brute	-1	5	0.757	19.688178	0.462467	0.882	0.145274	0.145314	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	2.213675	0.038159	NaN	...	brute	1	100	0.882	0.367369	0.009581	1.000	6.025749	6.027798	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	0.020864	0.002022	NaN	...	brute	1	100	1.000	19.904517	0.004004	0.757	0.001048	0.001048	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2d5f2b2c02d77766434d9e5033b4a76c	2cf49829391aa7891e877f2ed070adf0	2.993470	0.054501	NaN	...	brute	-1	100	0.882	19.381075	0.256970	0.663	0.154453	0.154467	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	2.280282	0.030067	NaN	...	brute	1	5	0.757	0.263199	0.009533	1.000	8.663719	8.669400	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	0.020747	0.000457	NaN	...	brute	1	5	1.000	4.084575	0.129852	0.922	0.005079	0.005082	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	b79d63bb7670c492de5c3befac58fe29	2cf49829391aa7891e877f2ed070adf0	1.184285	0.008049	NaN	...	brute	1	1	0.663	4.320357	0.043528	0.929	0.274117	0.274131	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	1.733147	0.027726	NaN	...	brute	-1	1	0.896	0.266812	0.007035	1.000	6.495768	6.498026	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	0.005129	0.003518	NaN	...	brute	-1	1	1.000	4.094661	0.056834	0.922	0.001253	0.001253	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	22d5eac4a4146149b35eae6914921514	88843f54689e3271092f70126e1de585	2.847563	0.058336	NaN	...	brute	-1	5	0.922	4.219127	0.078700	0.896	0.674918	0.675035	0.026

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.241	0.0	-1	1	19.059	0.307	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.210	0.0	-1	5	0.341	0.014	1.000	0.033	0.033	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.001	6.699	0.0	1	100	19.087	0.290	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.016	0.0	-1	100	0.363	0.007	1.000	0.031	0.031	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.001	6.765	0.0	1	5	4.090	0.069	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.885	0.0	1	1	0.282	0.006	1.000	0.041	0.041	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.353	0.0	-1	1	4.233	0.053	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.001	0.323	0.0	-1	5	0.268	0.007	1.000	0.019	0.019	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.955	0.109	0.000	0.002	-1	1	0.344	0.006	1.000	5.685	5.686	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.003	0.000	0.022	-1	1	19.093	0.077	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.860	0.071	0.000	0.003	-1	5	19.688	0.462	0.882	0.145	0.145	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.003	0.000	0.024	-1	5	0.348	0.014	1.000	0.070	0.070	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.214	0.038	0.000	0.002	1	100	0.367	0.010	1.000	6.026	6.028	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.002	0.000	0.021	1	100	19.905	0.004	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.993	0.055	0.000	0.003	-1	100	19.381	0.257	0.663	0.154	0.154	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.003	0.000	0.026	-1	100	0.334	0.013	1.000	0.077	0.077	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.280	0.030	0.000	0.002	1	5	0.263	0.010	1.000	8.664	8.669	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.000	0.000	0.021	1	5	4.085	0.130	0.922	0.005	0.005	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.184	0.008	0.001	0.001	1	1	4.320	0.044	0.929	0.274	0.274	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.001	0.000	0.019	1	1	0.279	0.007	1.000	0.070	0.070	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.733	0.028	0.000	0.002	-1	1	0.267	0.007	1.000	6.496	6.498	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.005	0.004	0.000	0.005	-1	1	4.095	0.057	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.848	0.058	0.000	0.003	-1	5	4.219	0.079	0.896	0.675	0.675	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.002	0.000	0.006	-1	5	0.284	0.005	1.000	0.021	0.021	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.850513	1.064081	NaN	...	kd_tree	-1	1	0.929	2.912771	0.234533	1.000	0.291994	0.292939	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.002804	0.000760	NaN	...	kd_tree	-1	1	1.000	127.467108	0.000000	0.946	0.000022	0.000022	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	dc74b969bf622bc24ba3bc62c980983b	1eb6cf1a720a225efb91df0b529b0510	1.079550	0.386281	NaN	...	kd_tree	-1	5	0.946	132.711713	0.000000	0.951	0.008135	0.008135	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	5.637007	0.623787	NaN	...	kd_tree	1	100	0.951	3.015049	0.230952	1.000	1.869624	1.875101	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	0.003605	0.000913	NaN	...	kd_tree	1	100	1.000	126.783379	0.000000	0.946	0.000028	0.000028	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	022f7445d43bb1dbc24dc3106c03cb93	1eb6cf1a720a225efb91df0b529b0510	3.259656	0.234377	NaN	...	kd_tree	-1	100	0.951	128.190496	0.000000	0.929	0.025428	0.025428	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	1.595688	0.171253	NaN	...	kd_tree	1	5	0.946	0.006028	0.000248	1.000	264.712076	264.935520	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	0.001597	0.000426	NaN	...	kd_tree	1	5	1.000	0.046574	0.012624	0.911	0.034296	0.035533	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	a9455565db1d8e052a783317c99744ff	1eb6cf1a720a225efb91df0b529b0510	0.939030	0.361407	NaN	...	kd_tree	1	1	0.929	0.075308	0.015822	0.894	12.469257	12.741481	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.029046	0.015580	NaN	...	kd_tree	-1	1	0.891	0.006052	0.000283	1.000	4.799086	4.804328	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.002461	0.000477	NaN	...	kd_tree	-1	1	1.000	0.043700	0.000469	0.911	0.056320	0.056323	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	dc74b969bf622bc24ba3bc62c980983b	edf7cb014c2c0910e1c3aaaf8ae633f2	0.024962	0.001064	NaN	...	kd_tree	-1	5	0.911	0.043073	0.000630	0.891	0.579517	0.579578	0.020

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	2.982	0.061	0.027	0.0	-1	1	127.529	0.000	0.929	0.023	0.023	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.731	0.078	0.021	0.0	-1	5	2.963	0.228	1.000	1.259	1.263	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.876	0.067	0.021	0.0	1	100	134.116	0.000	0.951	0.029	0.029	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.019	0.131	0.020	0.0	-1	100	2.975	0.196	1.000	1.351	1.354	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.941	0.122	0.020	0.0	1	5	0.048	0.012	0.891	82.946	85.341	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.780	0.108	0.021	0.0	1	1	0.006	0.000	1.000	662.469	662.895	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.018	0.0	-1	1	0.071	0.004	0.894	0.013	0.013	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.026	0.0	-1	5	0.006	0.000	1.000	0.104	0.104	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.851	1.064	0.000	0.001	-1	1	2.913	0.235	1.000	0.292	0.293	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	1	127.467	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.080	0.386	0.000	0.001	-1	5	132.712	0.000	0.951	0.008	0.008	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	3.116	0.211	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.637	0.624	0.000	0.006	1	100	3.015	0.231	1.000	1.870	1.875	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	0.000	0.004	1	100	126.783	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.260	0.234	0.000	0.003	-1	100	128.190	0.000	0.929	0.025	0.025	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.002	0.000	0.005	-1	100	2.947	0.222	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.596	0.171	0.000	0.002	1	5	0.006	0.000	1.000	264.712	264.936	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	0.047	0.013	0.911	0.034	0.036	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.939	0.361	0.000	0.001	1	1	0.075	0.016	0.894	12.469	12.741	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.006	0.000	1.000	0.172	0.172	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.029	0.016	0.001	0.000	-1	1	0.006	0.000	1.000	4.799	4.804	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	0.044	0.000	0.911	0.056	0.056	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.025	0.001	0.001	0.000	-1	5	0.043	0.001	0.891	0.580	0.580	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.006	0.000	1.000	0.363	0.363	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	25b9f14bed7dd5d830c6ccd4dfebbf0c	8c8fffa8ec4b2d8de833421f9e32beab	0.129184	0.005887	300	...	100	100	10	0.824	0.435038	0.013749	1.0	0.296949	0.297098	0.176

1 rows × 25 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	107.252	0.0	300	0.001	0.001	0.527	0.026	0.824	203.497	203.744	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.129	0.006	300	0.006	0.0	0.435	0.014	1.0	0.297	0.297	See	See